Simultaneous analysis of contours and durations

Functional PCA

Michele Gubian

July 1, 2024

Tutorial overview

  • Part 1: Functional PCA (1.5 hours)
  • Break (30 min)
  • Part 2: Landmark registration (1.5 hours)
  • Each part: Theory + Code
  • Questions: after Theory, after Code
  • All materials available on GitHub

Motivation

f0 contours and (syllable) boundaries

f0 contours and (syllable) boundaries

Goals

  • Anchor across-curves variation to meaningful boundaries (landmarks)
  • Separate variation due to:
    1. boundary misalignment
    2. shape variation anchored to boundaries
  • Analyse both sources of variation jointly

Predicted curves and boundary positions

Predicted curves and durations

Time-aligned curves

Interval durations

A suite of methods

  • Functional PCA
  • Landmark Registration
  • Linear (Mixed Effects) Regression

Applications (beyond f0 contours)

  • f0 and intensity
  • formants
  • multiple articulatory tracks (EMA)
  • any time-varying (bundle of) measurement …
  • … with or without boundary information

Examples

Code

  • based on R
  • recent R libraries: funData, MFPCA
  • based on the older fda
  • a (work in progress) library written by me: landmarkregUtils
  • all available on this GitHub repo

Arithmetic with functions

From time samples to functions

Sampled curve

Function \(f(t)\)

Addition

\(f(t) + \color{red}{g(t)}\)

Addition

\(f(t) + \color{red}{g(t)} = \color{blue}{y(t)}\)

Multiplication by a scalar

\(\color{red}{0.5} \cdot f(t)\)

Multiplication by a scalar

\(\color{red}{0.5} \cdot f(t) = \color{blue}{y(t)}\)

Multiplication by a scalar

\(\color{red}{(-0.5)} \cdot f(t) = \color{blue}{y(t)}\)

Mean curve

\(\frac{1}{2} \cdot (f(t) + \color{red}{g(t)})\)

Mean function

\(\frac{1}{2} \cdot (f(t) + \color{red}{g(t)}) = \color{blue}{y(t)}\)

Uni-dimensional curves

A curve dataset

100 curves, two categories

Procedure overview

Functional PCA

Procedure overview

A curve dataset

100 curves, remove factor information

Subtract mean curve

Mean \(\mu(t)\)

\(f_i(t) - \mu(t)\)

FPCA outcome

  • the mean curve \(\mu(t)\)
  • a number of Principal Component curves, \(PC1(t), PC2(t), \cdots\)
  • a set of corresponding scores, \(s_1, s_2, \cdots\) different for each curve

FPC curves

FPC scores

curveId Category s1 s2 s3
1 A -0.21 0.21 -0.02
2 A -0.20 -0.07 0.02
3 A -0.15 0.16 0.00
51 B 0.21 0.04 0.05
52 B 0.24 0.14 -0.03
53 B 0.27 0.17 0.06

FPC-based curve reconstruction

\(\color{blue}{\hat{f}(t)} = \mu(t)\)

FPC-based curve reconstruction

\(\color{blue}{\hat{f}(t)} = \mu(t) + \color{red}{(-0.21)} \cdot PC1(t)\)

FPC-based curve reconstruction

\(\color{blue}{\hat{f}(t)} = \mu(t) + \color{red}{(-0.21)} \cdot PC1(t) + \color{red}{0.21} \cdot PC2(t)\)

FPC-based curve reconstruction

\(\color{blue}{\hat{f}(t)} = \mu(t) + \color{red}{(-0.21)} \cdot PC1(t) + \color{red}{0.21} \cdot PC2(t) + \color{red}{(-0.02)} \cdot PC3(t)\)

FPC-based curve reconstruction

\(\color{blue}{\hat{f}(t)} = \mu(t)\)

FPC-based curve reconstruction

\(\color{blue}{\hat{f}(t)} = \mu(t) + \color{red}{0.21} \cdot PC1(t)\)

FPC-based curve reconstruction

\(\color{blue}{\hat{f}(t)} = \mu(t) + \color{red}{0.21} \cdot PC1(t) + \color{red}{0.04} \cdot PC2(t)\)

FPC-based curve reconstruction

\(\color{blue}{\hat{f}(t)} = \mu(t) + \color{red}{0.21} \cdot PC1(t) + \color{red}{0.04} \cdot PC2(t) + \color{red}{0.05} \cdot PC3(t)\)

FPC scores controlling FPC curves

FPCA summary

  • Input curves can be approximated by composing mean curve and PC curves multiplied by respective PC scores
  • PC scores are independent, i.e. if we know \(s_1\) we cannot guess \(s_2\)
  • All this is true also for other methods, e.g. DCT, Legendre polynomials, growth curves, etc.
  • But FPCA constructs PC curves optimised for your dataset, which makes the composition much more compact
  • In depth on inner workings of FPCA and other functional decomposition methods in Day 1 video of the previous version of this tutorial

Procedure overview

Reintroduce factors

Fit a regression model on s1

mod <- lm(s1 ~ Category, data = PCscores)
summary(mod)

Call:
lm(formula = s1 ~ Category, data = PCscores)

Residuals:
      Min        1Q    Median        3Q       Max 
-0.137063 -0.031294 -0.004309  0.032838  0.119646 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) -0.198269   0.007122  -27.84   <2e-16 ***
CategoryB    0.395197   0.010072   39.24   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.05036 on 98 degrees of freedom
Multiple R-squared:  0.9402,    Adjusted R-squared:  0.9395 
F-statistic:  1540 on 1 and 98 DF,  p-value: < 2.2e-16

Model predictions on s1

library(emmeans)
emmeans(mod, pairwise ~ Category)
$emmeans
 Category emmean      SE df lower.CL upper.CL
 A        -0.198 0.00712 98   -0.212   -0.184
 B         0.197 0.00712 98    0.183    0.211

Confidence level used: 0.95 

$contrasts
 contrast estimate     SE df t.ratio p.value
 A - B      -0.395 0.0101 98 -39.237  <.0001

Procedure overview

Reconstruct predicted curves from s1

\[ f_A(t) = \mu(t) + s_{1, A} \cdot PC1(t)\] \[ f_B(t) = \mu(t) + s_{1, B} \cdot PC1(t)\]

Reconstruct predicted curves from s1

Outcome

  • Compact model of modes of shape variations
  • Compact parametrisation of each curve in the dataset
  • Regression model predicting curve shapes from factors

GAM

Procedure overview

GAM Outcome

  • Compact model of modes of shape variations
  • Compact parametrisation of each curve in the dataset
  • Regression model predicting curve shapes from factors

GAM Outcome

Alternative workflows

Processing PC scores

  • If you have categories:
    • CART, Random forest, etc.
  • If you don’t have categories:
    • clustering

Proportion of variance

  • PCs are ordered by proportion of explained variance in the curve dataset
  • In our example, PC1 explains 51% of var., PC2 35%, etc.
  • This variance has nothing to do with the categories
  • A linear regression model s1 ~ Category explains 94% of the variance
  • A linear regression model s2 ~ Category explains 1% of the variance
  • This variance is the capacity of categories to predict a score
  • These two types of variance have nothing to do with each other!

Multi-dimensional curves

A 2-dimensional curve dataset

100 pairs of curves, two categories

D-dimensional Principal Components

  • There are D mean curves
    • \(\mu_{y1}(t)\) and \(\mu_{y2}(t)\)
  • Each PC is composed of D curves
    • \(PC1_{y1}(t)\) and \(PC1_{y2}(t)\), \(PC2_{y1}(t)\) and \(PC2_{y2}(t)\), etc.
  • Scores do not split in D
  • Each score controls a PC, which is made of D 1-dimensional curves
    • \(s_1\) controls (i.e. multiplies) \(PC1_{y1}(t)\) and \(PC1_{y2}(t)\)

Principal Components (PCs)

PC scores v. Category

Fit a regression model on s1

mod <- lm(s1 ~ Category, data = PCscores)
emmeans(mod, pairwise ~ Category)
Category \(s_1\)
A 0.134
B -0.134

Reconstruct predicted curves from s1

\[ y1_A(t) = \mu_{y1}(t) + \color{red}{s_{1, A}} \cdot PC1_{y1}(t)\] \[ y2_A(t) = \mu_{y2}(t) + \color{red}{s_{1, A}} \cdot PC1_{y2}(t)\]

\[ y1_B(t) = \mu_{y1}(t) + \color{red}{s_{1, B}} \cdot PC1_{y1}(t)\] \[ y2_B(t) = \mu_{y2}(t) + \color{red}{s_{1, B}} \cdot PC1_{y2}(t)\]

Reconstruct predicted curves from s1

Comments

  • Contours are multidimensional
  • PC scores behave as in uni-dimensional case
  • Flexible and computationally cheap
  • Difficult to obtain the same result with GAMs

References

Asano, Yuki, and Michele Gubian. 2018. “‘Excuse Meeee!!’:(mis) Coordination of Lexical and Paralinguistic Prosody in L2 Hyperarticulation.” Speech Communication 99: 183–200.
Asano, Yuki, Michele Gubian, and Sacha Dominik. 2016. “Cutting down on Manual Pitch Contour Annotation Using Data Modelling.”
Cronenberg, Johanna, Michele Gubian, Jonathan Harrington, and Hanna Ruch. 2020. “A Dynamic Model of the Change from Pre-to Post-Aspiration in Andalusian Spanish.” Journal of Phonetics 83: 101016.
El Zarka, Dina, Anneliese Kelterer, Michele Gubian, and Barbara Schuppler. 2024. “The Prosody of Theme, Rheme and Focus in Egyptian Arabic: A Quantitative Investigation of Tunes, Configurations and Speaker Variability.” Speech Communication, 103082.
Gubian, Michele, Lou Boves, and Francesco Cangemi. 2011. “Joint Analysis of f 0 and Speech Rate with Functional Data Analysis.” In 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 4972–75. IEEE.
Gubian, Michele, Francesco Cangemi, and Lou Boves. 2010. “Automatic and Data Driven Pitch Contour Manipulation with Functional Data Analysis.”
Gubian, Michele, Jonathan Harrington, Mary Stevens, Florian Schiel, and Paul Warren. 2019. “Tracking the New Zealand English near/square Merger Using Functional Principal Components Analysis.” In Proceedings of the 20th Annual Conference of the International Speech Communication Association, 296–300. Graz. https://doi.org/10.21437/Interspeech.2019-2115.
Gubian, Michele, Manfred Pastätter, and Marianne Pouplier. 2019. “Zooming in on Spatiotemporal v-to-c Coarticulation with Functional PCA.” In INTERSPEECH, 889–93.
Gubian, Michele, Francisco Torreira, and Lou Boves. 2015. “Using Functional Data Analysis for Investigating Multidimensional Dynamic Phonetic Contrasts.” Journal of Phonetics 49: 16–40. https://doi.org/10.1016/j.wocn.2014.10.001.
Ramsay, J., and B. Silverman. 2005. Functional Data Analysis. Springer Series in Statistics.